For a parallel distributed processing system in which multiples nodes (also referred to as “servers”, below) execute calculation in parallel, a network management technology has been known that secures both wide bandwidth and low latency for signal transmission of all-to-all communication (see, for example, Patent Document 1).